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The  DoD  acquisition 
process,  of  which  T&E 
is  a  part,  is  in  essence 
a  partnership  between 
government  and 
private  industry  that  has 
the  warfighter  at  its  heart 
and  as  its  main  concern. 


Dr.  Charles  E.  McQueary  has  an  extensive  back¬ 
ground  in  defense  systems  combined  with  a 
doctorate  in  mechanical  engineering.  His  in¬ 
volvement  with  the  National  Security  Industrial 
Association  and  the  defense  community  has 
given  him  a  first-hand  familiarity  with  both  security  issues 
and  technologies.  Set  to  retire  from  the  private  sector,  he 
instead  took  the  position  as  the  first  under  secretary  for 
the  Department  of  Homeland  Security’s  Science  and  Tech¬ 
nology  Directorate  in  2003.  On 
July  27,  2006,  McQueary 
was  sworn  in  as  director 
of  operational  test  and 
evaluation  (OT&E). 
Defense  AT&L  inter- 
^  !  viewed  McQueary  in 
October  2007  to  hear 
his  views  on  how  the 
test  and  evaluation 


community  is  working  to  better  serve  the  warfighter’s 
needs. 

Q 

You  became  the  director  of  OT&E — operational  test  and 
evaluation — in  July  of 2006.  Can  you  give  us  an  overview 
of  the  major  roles  and  responsibilities  of  your  new  posi¬ 
tion? 

A 

As  the  director,  I  am  responsible  to  the  secretary  of  de¬ 
fense  and  to  Congress  for  determining  that  operational 
testing  is  properly  planned  and  adequately  conducted, 
and  for  determining  the  operational  effectiveness  and 
suitability  of  a  system  for  carrying  out  the  intended  mis¬ 
sions  of  the  warfighter.  Operational  effectiveness  signifies 
the  level  of  mission  accomplishment  when  a  new  system 
is  employed  by  typical  users  in  the  planned  combat  envi¬ 
ronment.  Operational  suitability  is  the  level  of  system  reli¬ 
ability,  availability,  and  maintainability  achieved  in  order 
to  support  use  when  needed. 

I  am  also  responsible  for  oversight  and  test  plan  approval 
for  Live  Fire  T&E  for  the  department,  and  for  reviewing 
and  making  recommendations  to  the  secretary  of  defense 
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on  all  budgetary  and  financial  matters  relating  to  opera¬ 
tional  test  and  evaluation,  including  operational  test  facili¬ 
ties  and  equipment. 

Q 

The  OT&E  is  called  “ the  key  to  weapons  that  work.  ”  Given 
the  current  operations  tempo ,  your  mission  is  critical.  Can 
you  give  us  some  examples  of  how  your  department  is 
supporting  the  warfighter? 

A 

My  staff  is  closely  engaged  with  the  program  managers  of 
the  major  defense  acquisition  programs  most  critical  to 
our  warfighters,  such  as  mine-resistant  ambush-protected 
(MRAP)  vehicles.  Our  responsibility  is  to  make  certain  that 
the  systems  work  as  intended  by  demonstrating  mission 
performance  in  realistic  environments  prior  to  delivery 
to  the  warfighter.  These  demonstrations  take  place  on 
a  continual  basis  as  the  Service  operational  test  agen¬ 
cies,  staffed  by  military  personnel  who  were  warfighters 
in  previous  assignments,  conduct  early  assessments  on 
systems  in  development.  This  provides  a  user’s  perspec¬ 
tive  on  how  the  developing  systems  are  likely  to  work, 
identifies  risk  areas,  and  informs  the  decision  makers  who 
approve  the  systems’  progression  through  development 
and  demonstration. 

As  the  developing  systems  mature,  my  staff  works  with 
the  Service  acquisition  officials  and  operational  test  agen¬ 
cies  to  influence  the  planning  of  adequate  operational  test 
and  evaluation  prior  to  the  full-rate  production  decisions. 
As  the  Services  pursue  more  creative  and  agile  acquisition 
approaches,  especially  for  software-intensive  systems,  my 
staff  works  with  them  to  adapt  the  operational  test  and 
evaluation  strategies  so  that  “weapons  that  work”  are  de¬ 
livered  to  our  warfighters  on  time. 

OT&E  also  provides  support  to  combatant  command¬ 
ers  and  deployed  forces  in  Iraq  and  Afghanistan  through 
programs  such  as  our  information  assurance/information 
operations  training  and  assessment  program,  munitions 
effectiveness/explosives  in-theater  assessments,  Joint 
IED— improvised  explosive  devices— Defeat  Organization 
test  efforts,  and  body  armor  assessment  testing. 

Other  examples  of  direct  support  to  the  warfighter:  Our 
Center  for  Countermeasures  has  conducted  countermea¬ 
sures  training  for  deploying  forces  through  individual  Ser¬ 
vice  and  joint  training  exercises;  and  our  Joint  Test  and 
Evaluation  Program  has  developed  and  provided  stream¬ 
lined  tactics,  techniques,  and  procedures  in  the  areas  of 
force  protection,  enhanced  communications,  weapon  sys¬ 
tem  employment,  command  and  control,  and  logistics. 

Q 

One  of  the  priorities  of  OTSlE  is  to  conduct  test  and  evalu¬ 
ation  of  rapid  materiel  equipping  initiatives.  What  kind 


of  challenges  do  these  short  timeframes  and  rapid  turn¬ 
arounds  present  to  the  testing  community? 

A 

The  basic  challenge  to  the  OT&E  community  of  the  short¬ 
ened  timelines  involved  in  rapid  acquisition  and  urgent 
operational  requirements  is  to  provide  the  warfighters 
enough  information  on  the  systems  so  that  they  can  em¬ 
ploy  them  as  quickly  and  with  as  much  confidence  as 
possible.  That  is  critical;  if  a  system  can’t  be  counted  on  to 
perform  when  needed,  not  only  is  mission  success  jeop¬ 
ardized,  but  our  warriors  will  develop  doubts  about  the 
weapon  system’s  performance,  which  can  impact  both 
individual  and  organizational  mission  performance. 

One  response  to  the  challenge:  The  OT&E  community 
sometimes  worked  around  the  clock  at  ranges  such  as 
Yuma  and  Aberdeen  to  provide  24-  or  48-hour  turn¬ 
arounds  for  information  on  critical  equipment  and  sys¬ 
tems,  such  as  body  and  vehicle  armor. 

Testers  are  also  working  to  help  meet  the  urgent  needs 
of  our  warfighters  in  the  critical  mission  of  defeating 
IEDs.  The  Army  Test  &  Evaluation  Command  (ATEC)  has 
taken  on  the  mission  to  plan,  conduct,  and  report  the 
results  of  tests,  simulations,  experiments,  and  evalua¬ 
tions  to  ensure  our  warfighters  have  the  right  capabili¬ 
ties  for  success  across  the  entire  spectrum  of  operations. 
As  part  of  these  efforts,  testers  at  ATEC  are  conducting 
rapid  testing  in  direct  support  of  the  warfighter  to  pro¬ 
vide  information  on  the  capabilities  and  limitations  of 
untested  weapon  systems  issued  directly  to  our  soldiers 
conducting  combat  operations.  The  Joint  IED  Defeat 
Organization  expects  testers  to  use  flexible,  streamlined, 
and  tailored  test  procedures  based  on  standard  test  pro¬ 
tocols.  That  includes  reusing  knowledge  and  data  from 
other  projects;  sharing  data  among  Services  and  agen¬ 
cies;  and  providing  concise  and  timely  reports  to  enable 
decisions  on  fielding,  improvement,  or  termination. 

Like  the  Army,  the  Air  Force  T&E  community  is  working 
hard  to  be  responsive  to  the  urgent  operational  needs 
of  our  warfighters  and  is  providing  rapid  evaluations  of 
components  for  urgently  needed  capabilities  such  as  inte¬ 
grated  base  defense  security,  and  Global  Hawk  and  small 
diameter  bomb  employment. 

The  Navy  T&E  community  response  across  the  entire 
spectrum  of  urgently  needed  warfighter  capabilities  in¬ 
cludes  efforts  to  evaluate  and  provide  information  on  the 
Counter-Bomb/Counter-Bomber  Advanced  Concept  Tech¬ 
nology  Demonstration,  which  will  help  meet  evolving, 
asymmetrical,  and  sophisticated  terrorist  threats.  These 
detection  and  mitigation  systems  will  provide  force  pro¬ 
tection  personnel  with  the  latest  concept  of  operations, 
tactics,  techniques,  and  procedures;  and  with  rules  of  en¬ 
gagement  generation,  update,  and  dissemination. 
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Dr.  Charles  E.  McQueary 


Director  of  Operational  Test  and 
Evaluation 

r.  Charles  E.  McQueary 
was  sworn  in  as  director 
of  operational  test  and 
evaluation  on  July  27,  2006. 

A  presidential  appointee 
confirmed  by  the  U.S.  Senate, 
he  serves  as  the  senior  advisor 
to  the  secretary  of  defense 
on  testing  Department  of 
Defense  weapon  systems, 
and  prescribing  policies  and 
procedures  for  the  conduct  of 
operational  and  live  fire  test 
and  evaluation. 

Prior  to  his  current  ap¬ 
pointment,  McQueary  was 
confirmed  by  the  U.S.  Senate  in  March  2003  as  the  first  under 
secretary  for  science  and  technology  at  the  Department  of 
Homeland  Security.  In  that  position,  he  led  the  research  and 
development  arm  of  DHS,  utilizing  the  nation's  scientific  and 
technological  resources  to  provide  federal,  state,  and  local 
officials  with  the  technology  and  capabilities  to  protect  the 
homeland. 

McQueary  is  a  former  president  of  General  Dynamics 
Advanced  Technology  Systems,  in  Greensboro,  N.C.  He  has 
also  been  president  and  vice  president  of  business  units  for 
AT&T  and  Lucent  Technologies,  and  a  director  for  AT&T  Bell 
Laboratories. 

Early  in  his  career  at  Bell  Laboratories,  McQueary  served 
as  head  of  the  Missile  Operations  Department  for  the  SAFE¬ 
GUARD  Antiballistic  Missile  Test  Program,  based  at  Kwajalein 
in  the  Marshall  Islands.  He  later  headed  Bell  Laboratories' 
Field  Operations  Department  in  Great  Britain  in  support  of  a 
Navy  oceanographic  research  station.  He  also  served  as  the 
director  of  the  Undersea  Systems  Development  Lab. 

McQueary  is  a  former  executive  board  member  of  the 
National  Security  Industrial  Association  and  the  American 
Defense  Preparedness  Association  (both  later  combined  to 
form  the  National  Defense  Industrial  Association).  He  is  a  past 
chairman  of  the  Undersea  Warfare  Systems  Division  of  the 
American  Defense  Preparedness  Association,  and  a  former 
member  of  the  Navy  League  Industrial  Executive  Board,  the 
Navy  Submarine  League,  the  Electronics  Industries  Associa¬ 
tion,  the  American  Society  of  Mechanical  Engineers,  and  the 
American  Association  for  the  Advancement  of  Science.  He  is 
also  the  recipient  of  the  National  Defense  Industrial  Associa¬ 
tion  Homeland  Security  Leadership  Award. 

A  native  of  Texas,  McQueary  is  a  graduate  of  The  Univer¬ 
sity  of  Texas,  Austin,  where  he  earned  a  bachelor's  and  a 
master's  degree  in  mechanical  engineering,  and  a  doctorate 
in  engineering  mechanics,  the  latter  two  as  a  NASA  Scholar. 
He  is  a  member  of  five  academic  honor  societies.  The  Univer¬ 
sity  of  Texas  has  named  McQueary  a  Distinguished  Engineer¬ 
ing  Graduate. 


Another  way  that  testers  are  meeting  the  chal¬ 
lenge  of  shortened  timelines  is  to  work  in  theater 
to  ensure  that  information  needs  are  identified 
to  stateside  test  ranges  and  that  the  information 
supplied  from  the  ranges  is  relevant  and  properly 
reported  to  our  forces.  This  response  by  the  T&E 
community  is  especially  noteworthy  because  it 
often  involves  a  level  of  commitment  and  sac¬ 
rifice  that  is  outside  the  typical  test  and  evalua¬ 
tion  operating  envelope.  It  exemplifies  the  ends 
to  which  the  T&E  community  is  prepared  to  go 
to  support  our  warfighters. 

Q 

You  have  put  a  great  emphasis  on  reaching  out  to 
academic  and  scientific  communities  and  on  build¬ 
ing  bridges  with  private  enterprise.  What  types  of 
benefits  accrue  from  these  relationships?  What 
are  you  doing  to  encourage  continued  communi¬ 
cation? 

A 

I  believe  that  communication  between  govern¬ 
ment  and  private  industry  is  absolutely  critical 
so  we  can  jointly  review  issues  of  common  inter¬ 
est  and  concern  and  review  those  encompassing 
T&E  policies  and  procedures  that  impact  weap¬ 
ons  systems  development,  procurement,  and 
use.  This  is  critical  because  the  DoD  acquisition 
process,  of  which  T&E  is  a  part,  is  in  essence 
a  partnership  between  government  and  private 
industry  that  has  the  warfighter  at  its  heart  and 
as  its  main  concern.  We  need  to  communicate 
and  understand  each  other  so  we  can  work  as 
smoothly  and  efficiently  as  possible  in  support 
of  our  warfighters. 

To  maintain  and  participate  in  this  dialogue  with 
private  industry,  OT&E  participates  in  several 
government/private  industry  forums.  One  such 
forum  is  the  National  Defense  Industrial  Asso¬ 
ciation’s  Industrial  Committee  on  Operational 
Test  and  Evaluation,  or  the  ICOTE.  The  ICOTE  is 
chaired  by  Larry  Graviss  (a  representative  from 
private  industry)  and  meets  four  times  a  year  to 
discuss  important  and  emerging  T&E  policies  and 
issues.  OT&E  is  also  working  with  the  Govern¬ 
ment  Electronics  and  Information  Technology  As¬ 
sociation— a  standards  house— to  facilitate  a  new 
reliability  program  standard  that  will  assist  PMs 
in  setting  up  and  managing  effective  reliability 
programs,  based  upon  industry  best  practices. 

OT&E  also  participates  on  Defense  Science  Board 
task  forces  that  survey  private  industry  for  best 
practices  that  are  applicable  to  acquisition  and 
T&E.  We’ve  also  commissioned  reports  from  the 
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National  Academy  of  Sciences  on  how  private  industry 
conducts  T&E. 

Through  these  forums,  OT&E  maintains  close  commu¬ 
nication  with  private  industry  so  that  the  government/ 
private  industry  partnership  concerned  with  acquisition 
of  major  systems  for  our  warfighters  is  as  strong  and  as 
effective  as  possible. 

Q 

You’ve  stated  that  many  challenges ,  such  as  cost  growth 
and  technology  readiness  problems ,  could  be  greatly  re¬ 
duced  if  more  emphasis  was  placed  on  the  early  phases  of 
the  development  process.  Could  you  expand  on  that? 

A 

I  strongly  believe  that  OT&E  should  be  a  process  of  con¬ 
firmation  and  not  one  of  discovery.  Unfortunately,  OT&E 
is  too  often  the  place  where  performance  shortcomings 
and  new  failure  modes  are  discovered.  When  problems 
are  discovered  late  in  the  acquisition  process,  the  cost  to 
fix  these  problems  is  much  higher  than  if  they  were  dis¬ 
covered  earlier.  In  addition,  the  time  lost  when  problems 
are  found  at  this  stage  can  be  substantial— and  when  our 
forces  need  a  new  capability,  the  latter  penalty  may  be 
even  more  substantial  than  increased  cost. 

To  move  OT&E  into  the  role  of  confirmation  and  away 
from  the  role  of  a  discoverer  of  problems,  we  must  do 
two  things:  We  must  incorporate  operational  realism 
into  developmental  testing  to  gain  operational  insights 
and  identify  failure  modes  as  early  as  possible;  and  we 
must  infuse  sound  reliability  engineering  practices  into 
the  systems  engineering  and  developmental  testing  stage. 
Poor  reliability  drives  down  mission  accomplishment  and 
affordability  and  drives  up  force  structure,  total  cost  of 
ownership,  and  the  logistics  footprint. 

To  incorporate  operational  realism  into  developmental 
testing,  OT&E  is  working  with  the  Service  OTA — opera¬ 
tional  test  authority— commanders  who  share  this  goal. 
And  to  infuse  sound  reliability  engineering  practices  into 
the  systems  engineering  and  developmental  testing  pro¬ 
cess,  OT&E  has  worked  with  the  under  secretary  of  de¬ 
fense  for  acquisition,  technology  and  logistics  to  update 
the  DoD  RAM  [Reliability  Availability  and  Maintainability ] 
Handbook,  and  to  write  and  submit  a  congressionally 
mandated  report  on  current  T&E  policies  and  practices 
and  how  they  can  be  improved.  To  further  pursue  the 
goal  of  increased  reliability,  OT&E  and  OUSD(AT&L)  are 
jointly  sponsoring  a  Defense  Science  Board  Task  Force 
that  will  examine  how  to  strengthen  developmental  T&E 
oversight,  and  we  are  working  together  to  provide  some 
guidance  for  the  development  of  the  mandatory  key  per¬ 
formance  parameters  for  materiel  availability.  OT&E  and 
OUSD(AT&L)  are  also  working  with  the  Joint  Staff  on  a 
method  for  developing  and  justifying  reliability,  availabil¬ 


ity,  and  maintainability  requirements,  and  looking  at  the 
system  development  contracting  process— specifically  the 
request  for  proposal  to  industry— to  determine  how  to  in¬ 
clude  reliability  requirements  in  the  system  development 
statement  of  work. 

Q 

A  recent  study  commissioned  by  your  department  looked 
at  the  empirical  relationships  between  reliability  invest¬ 
ment  and  life-cycle  support  costs.  It  suggests  a  relationship 
between  achieved  reliability  improvement  and  reduction 
in  overall  support  costs.  Is  there  a  growing  emphasis  on 
establishing  reliability  goals  and  reliability  improvements 
into  programs?  How  might  this  focus  affect  OT&E  opera¬ 
tions? 

A 

DoD  is  placing  a  strong  emphasis  on  reliability  because  it 
directly  impacts  the  safety,  functionality,  and  cost  of  our 
systems.  This  is  evidenced  by  Joint  Requirements  Over¬ 
sight  Committee  approval  of  the  materiel  availability  key 
performance  parameter,  which  has  both  reliability  and 
ownership  costs  as  key  systems  attributes.  Test  results 
since  2001  show  that  almost  50  percent  of  DoD’s  pro¬ 
grams  in  oversight  are  unsuitable  at  the  time  of  initial 
operational  test  and  evaluation— IOT&E— because  they  do 
not  achieve  reliability  goals.  When  I  became  the  director 
of  OT&E,  I  made  system  suitability  the  number  one  prior¬ 
ity  because  reliability  is  a  key  enabler  of  suitability,  and  it 
directly  impacts  design  and  development  costs,  support 
costs,  logistics  footprint,  and  system  downtime. 

In  this  vein,  the  Joint  Staff  and  the  Office  of  the  Secretary 
of  Defense  are  sponsoring  a  number  of  initiatives  to  in¬ 
crease  emphasis  on  reliability.  The  study  you  referenced, 
which  showed  the  empirical  relationships  between  re¬ 
liability  investment  and  life-cycle  support  costs,  is  one 
of  these  initiatives  and  was  conducted  by  the  Logistics 
Management  Institute. 

Some  other  initiatives  are  developing  the  RAM  Rational 
Guide;  incorporating  reliability  metrics  into  the  major  de¬ 
fense  acquisition  program  oversight  process;  facilitating 
a  new  commercial/government  reliability  program  stan¬ 
dard  to  assist  PMs  in  setting  up  and  managing  effective 
reliability  programs;  and  developing  common  standards 
testing  for  the  OTAs  to  use  during  testing  for  sustainability 
and  reliability. 

The  expected  effect  on  OT&E  operations  will  be  a  greater 
percentage  of  programs  entering  IOT&E  that  have  dem¬ 
onstrated  robust  reliability  designs  and  planned  reliabil¬ 
ity  growth  during  developmental  testing.  Historical  data 
shows  that  nearly  60  percent  of  the  programs  that  pass 
developmental  testing  also  pass  OT&E.  We  expect  to  see 
an  increase  in  the  number  of  DoD  programs  successfully 
passing  IOT&E  the  first  time  around. 
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Q 

AFOTEC,  the  Air  Force  Operational  Test  and  Evaluation 
Center,  has  begun  to  implement  a  capability-based  rat¬ 
ing  system.  Can  you  describe  this  approach  and  how  it 
compares  to  the  effectiveness  and  suitability-based  rating 
system?  Is  it  a  more  responsive  approach? 

A 

I  view  the  AFOTEC  capability-based  rating  system  as  a 
well-intentioned  process  improvement  to  make  their 
products  more  meaningful  and  responsive  to  the  warf¬ 
ighter.  The  capability-based  approach  presented  assess¬ 
ments  at  a  higher  level  than  the  operational  effectiveness 
(can  warfighters  execute  their  mission  with  the  system?) 
and  suitability  (can  the  system  be  used  in  the  field  by 
typical  users?)  approaches.  The  premise  was  to  assess 
the  critical  operational  issues  to  ultimately  resolve  how 
capable  a  unit  equipped  with  the  system  under  test  is  of 
performing  its  intended  missions. 


This  discourse  allowed  us  to  move  further  away  from  the 
pass/fail  mentality  to  one  of  providing  independent  as¬ 
sessments  of  capabilities  and  limitations  of  the  systems 
we  test.  I  see  the  AFOTEC  initiative  as  a  good  example  of 
a  process  improvement  that  created  a  greater  dialogue 
and  understanding  with  the  developmental  testers  and 
acquisition  community,  allowing  us  to  present  our  best 
judgments  earlier  in  acquisition  to  the  capability  demon¬ 
strated  to  date  in  the  environments  to  which  the  system 
has  been  subjected. 

Q 

There  have  been  many  studies  that  look  at  how  commercial 
enterprises  conduct  testing  and  evaluation.  Industry  tends 
to  test  earlier  in  a  program's  development,  and  testing  is 
seen  as  an  integral  part  of  a  program's  success.  Is  DoD 
testing  and  evaluation  able  to  import  some  commercial 
best  practices? 

A 


I  strongly  believe  that  OT&E  should 
be  a  process  of  confirmation 
and  not  one  of  discovery.  When 
problems  are  discovered  late 
in  the  acquisition  process,  the 
cost  to  fix  these  problems  is 
much  higher  than  if  they  were 
discovered  earlier. 


Yes;  in  fact,  one  of  the  OSD  reliability-improvement  initia¬ 
tives  currently  under  way  is  to  facilitate  development  of  a 
new  commercial/government  reliability  program  standard 
within  the  Government  Electronics  Information  Technol¬ 
ogy  Association  standards  process.  Unlike  the  cancelled 
MIL-STD-785B,  which  was  not  effective  in  developing 
highly  reliable  systems,  this  standard  will  include  a  num¬ 
ber  of  commercial  best  practices.  In  addition  to  testing  the 
inherent  reliability  of  the  design  in  the  design  phases,  the 
new  standard  will  embrace  a  number  of  other  best  prac¬ 
tices  such  as  emphasizing  the  importance  of  thoroughly 
understanding  reliability  requirements  to  characterize  and 
shape  the  reliability  program;  and  ensuring  that  reliability 
objectives  are  an  integral  part  of  the  business  strategy  and 


The  AFOTEC’s  capability-based  approach  has  continued 
to  evolve  over  the  last  year  to  focus  more  on  earlier  opera¬ 
tional  testing  involvement  and  improved  suitability,  but 
it  engendered  a  greater  dialogue  between  all  the  opera¬ 
tional  testers,  improving  the  operational  testing  process. 
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have  demonstrated  commitment  of  senior  management, 
that  reliability  tasks  are  an  integral  part  of  the  systems  en¬ 
gineering  process  and  should  be  addressed  concurrently 
with  other  engineering  and  design  activities  beginning 
early  in  the  design  phase,  that  use  environment  and  duty 
cycles  along  with  their  related  stresses  must  be  under¬ 
stood  for  entire  life  cycle,  and  that  root-cause  analysis  of 
critical  failure  modes  must  be  accomplished  to  eliminate 
or  minimize  their  consequences. 

Q 

Historically,  there  has  often  been  an  adversarial  relation¬ 
ship  between  PMs  and  the  testing  community.  Test  scores 
have  been  perceived  as  scorecards  that  might  indicate 
whether  the  program  is  to  continue  or  to  receive  further 
funding.  A  new  emphasis  sees  testing  as  an  opportunity 
to  capture  knowledge  and  fix  problems  much  earlier  in  a 
program's  development.  Do  you  see  a  cultural  change  oc¬ 
curring?  Are  attitudes  shifting? 

A 

In  my  experience  as  the  director  of  OT&E,  I  would  certainly 
not  characterize  relationships  between  program  managers 
and  the  test  community  as  adversarial.  Although  I  can’t 
comment  on  the  historical  approaches  of  others  to  OT&E, 
my  experience  in  private  industry  reinforces  the  view  that 
successful  programmers  and  testers  always  find  a  way  to 
partner  in  pursuit  of  a  common  objective,  like  producing 
weapon  systems  that  work.  Those  who  have  participated 
in  the  acquisition  process  clearly  understand  there  will 
always  be  fundamental  challenges  associated  with  cost, 
schedule,  and  performance.  The  very  nature  of  those  pres¬ 
sures  almost  always  produces  friction.  Some  view  fric¬ 
tion  negatively;  my  personal  view  is  that  programmatic 
tension  tends  to  have  a  strengthening  effect  that  usually 
results  in  a  better  product. 

With  regard  to  the  impact  of  test  performance  on  fund¬ 
ing  decisions,  OT&E  is  very  careful  to  stay  in  its  lane.  Our 
mandate  is  to  provide  the  acquisition  decision  maker  our 
independent  assessment  of  a  system’s  operational  effec¬ 
tiveness  and  suitability  based  on  an  adequate  operational 
test.  I  believe  OT&E  does  this  very  well.  We  fully  under¬ 
stand  that  our  assessment  will  be  one  of  many  inputs  that 
acquisition  decision  makers  will  use  to  make  tough  calls 
in  an  environment  of  scarce  resources. 

As  you  mentioned,  there  may  also  be  a  new,  emerging 
dynamic  of  test-fix-test  based  on  meeting  urgent  field  re¬ 
quirements.  The  MRAP  testing  is  a  great  example.  As  the 
vehicles  go  through  developmental  testing  at  Aberdeen 
Test  Center,  soldiers  and  Marines  with  recent  combat  ex¬ 
perience  are  working  closely  with  industry  to  identify  op¬ 
erational  problems.  Industry  then  moves  immediately  to 
fix  or  change  such  items  as  door  locations,  hatches,  seats, 
gunner  cupolas,  and  internal  equipment.  So  industry  can 
use  a  pencil  eraser  on  an  MRAP  design  rather  than  re¬ 


engineer  an  entire  production  line.  The  cost  savings  and 
reduced  response  cycle  times  are  quite  evident. 

At  the  same  time,  it  is  probably  unrealistic  to  expect  this 
type  of  operational  involvement  on  every  system  under¬ 
going  developmental  testing.  There  are  simply  not  enough 
operators  to  spread  around  the  developmental  test  com¬ 
munity.  What  we  can  and  should  do  is  capture  the  lessons 
learned  from  a  fast-moving  program  like  MRAP  and  see  if 
we  can  apply  them  across  the  acquisition  community. 

Q 

How  is  OT&E  working  with  program  managers  in  the  early 
phases  of  a  program?  How  much  influence  should  OT&E 
have,  for  example,  in  the  initial  capability  document  de¬ 
sign?  Is  there  effective  communication  at  this  stage  in  the 
process? 

A 

We  work  with  program  management  right  from  the  start 
of  the  program.  The  acquisition  strategy  is  something  with 
which  we  must  concur  because  it  has  a  section  on  the 
T&E  strategy,  which  is  near  the  beginning  of  the  acquisi¬ 
tion  strategy — confirming  T&E’s  importance. 

Although  this  kind  of  formal  communication  is  effective, 
we  also  work  within  the  early  phases  of  a  program  in 
many  informal  ways.  An  example  of  this  is  how  we  work 
with  the  requirements  community  within  the  JCIDS  [Joint 
Capabilities  Integration  and  Development]  process  to  help 
define  requirements  in  a  meaningful,  testable  way. 

It  is  also  important  to  understand  that  OT&E  is  not  the 
only  operational  test  and  evaluation  influence.  The  Service 
OTAs  also  interact  and  have  some  influence  in  the  early 
phases  of  a  program.  The  example  they  would  probably 
feel  most  comfortable  with  is  their  commenting  on  the 
testability  of  a  requirement. 

To  be  a  bit  philosophical,  the  influence  we  should  have 
should  come  from  the  insight  testing  can  give  a  program 
on  its  progress  and  its  risk  areas.  To  increase  the  value  of 
this  kind  of  early  insight,  OT&E  is  working  with  the  Ser¬ 
vice  OTAs  to  increase  operational  realism  in  early  testing 
so  there  will  be  fewer  surprises  in  IOT&E. 

Q 

It  has  been  suggested  that  spiral  development  allows  for 
testing  much  earlier  in  a  program's  design,  instead  of  near 
the  end  of  development  as  has  been  the  case  in  previous 
years.  Do  you  see  this  type  of  testing  yielding  promising 
results? 

A 

Testing  early  in  the  program’s  design  phase  is  almost 
always  worth  it,  no  matter  what  you  call  the  design  pro¬ 
cess,  and  infusing  operational  realism  early  in  testing  only 
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makes  early  testing  more  valuable.  It’s  the  concept  that 
is  important,  not  what  you  call  it— especially  when  you 
consider  that  the  phase  “spiral  development”  is  scheduled 
to  be  dropped  from  the  Acquisition  Guidebook.  So  your 
point  that  “early  is  good”  is  the  concept  that  must  survive, 
and  not  necessarily  any  particular  name  or  phase  that  we 
may  use  to  describe  it. 

Q 

Are  there  other  topics  you'd  like  to  share  with  our  read¬ 
ers? 


Combat  Capability,  or  NECC,  have  led  us  to  adapt  and 
expand  proven  practices  such  as  our  Risk  Assessment 
Level  of  Test  process  to  better  meet  both  developer  and 
warfighter  needs.  The  RALOT  process  has  been  used  for 
several  years  to  assess  the  level  of  testing  needed  for  pe¬ 
riodic  software  updates  after  completion  of  a  program’s 
IOT&E.  I  recently  approved  the  piloting  of  a  new  approach. 
For  NECC,  the  Joint  System  Test  Team  will  use  the  RALOT 
process  prior  to  IOT&E  to  assess  the  level  of  testing  that 
is  appropriate  to  the  risk  associated  with  the  individual 
capability  module  being  developed. 


A 

I’d  like  to  talk  about  the  Department’s  transition  to  net- 
centric  systems  and  how  some  have  suggested  that  the 
current  testing  process  doesn’t  provide  the  agility  and 
flexibility  needed  to  keep  pace. 


At  the  same  time,  we  are  examining  the  results  of  our 
information  assurance  and  interoperability  assessments 
of  fielded  systems  to  identify  key  focus  areas  for  both 
system  developers  and  testers. 


Some  view  friction 
negatively;  my  personal 
view  is  that  programmatic 
tension  tends  to  have  a 
strengthening  effect  that 
usually  results  in  a  better 
product. 


The  new  approaches  being  employed  for  the  de¬ 
velopment  of  systems  such  as  the  Net-Enabled 


There  is  no  doubt  that  the  transition  to  net-centric  sys¬ 
tems  poses  new  challenges  for  the  acquisition  commu¬ 
nity.  Keeping  up  with  the  rapid  pace  of  development  in 
information  systems  and  with  the  complex  interaction  of 
the  multitude  of  systems  is  inherently  challenging.  We  are 
also  confronted  with  the  sometimes  competing  goals  of 
ensuring  our  information  systems  are  interoperable  with 
both  civil  and  international  partners  while  simultaneously 
assuring  their  security.  To  help  meet  this  challenge,  I  re¬ 
cently  established  a  new  position  for  a  deputy  director  for 
net-centric  and  space  systems.  This  raises  net-centrr 
space  systems  oversight  to  the  same  management 
as  air,  land,  and  naval  warfare. 


Realistic  testing  of  complex  networks  in  a  dynamic  envi¬ 
ronment  remains  a  challenge.  We  are  looking  to  partner, 
where  feasible,  with  the  training  community  to  leverage 
live,  virtual,  and  constructive  events  to  provide  the  most 
realistic  environment  for  assessing  the  operational  ef¬ 
fectiveness,  suitability,  and  survivability  of  these  critical 
combat  systems. 

Q 

Dr.  McQueary 
thank  you. 
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